Add pdd protocol and multistream example as .md files for easier review and upgrade collaboratively #12
Conversation
…ew and upgrade collaboratively
|
I still think there's confusion about the transition between On Thu, Jun 11, 2015 at 06:21:47AM -0700, David Dias wrote 1:
Looking through the referenced comment, it sounds like the goal is
Supporting mute ends in the protocol means that there is the sender We may also need to keep the multistream channel around (even if we → stream 0 /multistream/1.0.0 That way you know pairs that agree on a multistream version will be |
|
On Thu, Jun 11, 2015 at 09:24:55AM -0700, W. Trevor King wrote:
Ah, I left out “conectee is a speaker in a push-stream”. For the ← stream 0 /multistream/1.0.0 But and with a chatty connector that tries to re-negotiate the ← stream 0 /multistream/1.0.0 However, with a mute connector (e.g. you have a mute client that wants |
|
In #11 I suggested an ABNF-like syntax 1 for defining the protocol |
|
On Thu, Jun 11, 2015 at 10:15:37AM -0700, W. Trevor King wrote:
Likely these folks have some interesting papers 1, since they |
| > /dogs/0.1.0 | ||
| < /dogs/0.1.0 | ||
| > hey | ||
| ``` |
There was a problem hiding this comment.
ideally the test could try to fuzz all possible message orderings. e.g.:
# order 1
< /multistream/1.0.0
> /multistream/1.0.0
> ls
< ["/dogs/0.1.0","/cats/1.2.11"]
> /mouse/1.1.0
< na
> /dogs/0.1.0
< /dogs/0.1.0
> hey
# order 2
> /multistream/1.0.0
< /multistream/1.0.0
> ls
< ["/dogs/0.1.0","/cats/1.2.11"]
> /mouse/1.1.0
< na
> /dogs/0.1.0
< /dogs/0.1.0
> hey
# order 3
> /multistream/1.0.0
> ls
< /multistream/1.0.0
< ["/dogs/0.1.0","/cats/1.2.11"]
> /mouse/1.1.0
< na
> /dogs/0.1.0
< /dogs/0.1.0
> hey
... and so on down ...
# another (this one is weird from an "interactivity" perspective
# but --for correctness-- it should be tested to ensure things behave
# as expected).
> /multistream/1.0.0
> ls
> /mouse/1.1.0
> /dogs/0.1.0
> hey
< /multistream/1.0.0
< ["/dogs/0.1.0","/cats/1.2.11"]
< na
< /dogs/0.1.0
not sure if this is worth worrying about or not
There was a problem hiding this comment.
On Thu, Jun 11, 2015 at 08:55:04PM -0700, Juan Batiz-Benet wrote:
ideally the test could try to fuzz all possible message orderings…
How do you know what the possible message orderings are without a
protocol spec to tell you?
stream closing is a tricky problem and so i just left it out of multistream. in practice, Basically, to speak more than one protocol, the user must use a muxing protocol, even if that is sequential: Hypothetical sequential muxer: |
Eddie Kohler is one of the best people in networks/systems research. Not sure if he's still interested in all this stuff, but I'm sure he'd have a great perspective. I'll reach out. |
|
@wking you're basically totally right here. we should be doing this right. protocol compilation is absolutely the right way to think about this. My only worry is "how usable" are all these things. (i.e. in this specific case, how do we justify using "the right thing" when "worse is better". time is a huge factor in all these sorts of decisions). This tradeoff is a really hard question. Looking at massively successful systems (git, linux, go, js), none of them is fully the right thing and the practices were instead "good enough" for the most part (though with relentless testing in the git + linux side). I think @diasdavid's attempt is a pretty good start to something "good enough" for our use case here, so maybe we can use it as a very simple tool. I would be wary of making it much bigger without deeply understanding the relevant literature, understanding what's available already, and thinking from there. |
|
On Thu, Jun 11, 2015 at 09:12:51PM -0700, Juan Batiz-Benet wrote:
I just don't understand how the current suggestion is going to work. |
|
@wking I've added a more detailed description on the "## Protocol" segment, that I hope it clarifies how Stream Multiplexing is handled (which is not directly by multistream) and how multistream lives on top of the transport. As far for being ['silent', 'broadcast', 'interactive', 'select'], these modes come in pairs and the decision of how one of the ends operates comes beforehand. e.g: I will that tcp://time.service:8888 offers a service in broadcast multistream mode that gives me current time, so I know that I should connect silently. |
|
On Fri, Jun 12, 2015 at 06:21:13AM -0700, David Dias wrote:
Right. My issue here is that in order to test an implementation
For flexible-enough implementations, you may also have to pass that Thinking this over last night, I think you may be able to convey most |
Indeed, but is it really a issue? Most expectations library depart from the assumption that we "prepare a scenario" and "expect" some given results. I don't necessarily see that asking a implementation developer to recreate a scenario for a given test would be problem, but maybe I'm not seeing all the implications. For multistream case, it only expects a duplexStream interface, so there wouldn't be a need to use network even (which would make impl more resilient since they won't be able to make assumptions on the network stack). In the end, for multistream test, the question is "If I throw all of this messages at you, will you tell me what I'm expecting to hear?" kind of test.
This would be very smart! But then again means that multistream behaviour would be dependent of the underlying transport and with cases such as laying multistream on top of spdy or http/2, transports would have to be monkey patched to accept behave as multistream expects them too. |
|
On Fri, Jun 12, 2015 at 09:44:25AM -0700, David Dias wrote:
It's obviously not an insolvable problem ;), but with the current spec
What's a duplexStream interface? Are there C bindings? What if my
Is there no way to SHUT_WR (etc.) a particular virtual stream inside |
|
@wking i think this discussion is a huge distraction. I think what @diasdavid has in mind works well for now. |
| │ message length││ ││ | ||
| │└───────────────┘└───────┘ | ||
| ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ | ||
| ``` |
There was a problem hiding this comment.
one comment: we don't need this frame for every message. the multistream idea is just to make "multistream protocols" that all begin with the recognizable header, but after that could pipe your own thing. the reason for it is that sometimes protocols already have their own framing layer so dont want to waste too much on that.
(maybe you already mean this -- wasnt getting it from the above)
There was a problem hiding this comment.
On Fri, Jun 12, 2015 at 04:45:29PM -0700, Juan Batiz-Benet wrote:
one comment: we don't need this frame for every message. the
multistream idea is just to make "multistream protocols" that all
begin with the recognizable header…
That's enough for the test-suite's multistream-select and multistream
drivers (that get messages from the transcript and encode them
correctly for those protocols).
… but after that could pipe your own thing.
So the test suite will also need protocol drivers for any protocols
its testing. Each per-protocol driver is responsible for testing both
the syntactic and semantic correctness of the testee's implementation
of that protocol, and the interface for a per-protocol driver has to
accept messages that the test suite is reading from the transcript
(e.g. ‘hey, how is it going?’) and return messages that it receives
from the testee.
There was a problem hiding this comment.
one comment: we don't need this frame for every message.
... (maybe you already mean this -- wasnt getting it from the above)
Yes, I hope now it's more clear, once a protocol is handshake'd, it is up to that protocol to control the stream and encode however they want.
|
@diasdavid this LGTM |
| └────────────────────────────────┘ | ||
| ``` | ||
|
|
||
| multistream doesn't cover stream multiplexing over the same connection, however, we can achieve this by leveraging functionality offered by SPDY or HTTP/2. |
There was a problem hiding this comment.
So it should probably just be multistream (and not multistream-select in the figure above). Without a stream-multiplexing transport layer, using multistream-select will be difficult (impossible?).
There was a problem hiding this comment.
Without a stream-multiplexing transport layer, using
multistream-selectwill be difficult (impossible?).
no. it will not. easy, as the examples suggest. the stream will just be locked into that protocol until the stream terminates.
There was a problem hiding this comment.
On Sat, Jun 13, 2015 at 08:43:29PM -0700, Juan Batiz-Benet wrote:
Without a stream-multiplexing transport layer, using
multistream-selectwill be difficult (impossible?).no. it will not. easy, as the examples suggest stream will just be
locked into that protocol until the stream terminates.
Ah, I was just thinking that multistream-select assumed it would
always be running for stream maintenance. But looking over [1,2], it
looks like ‘multistream’ is just “write a protocol-describing header
line before sending any other data down the stream”. So that's fine.
Then ‘multistream-select’ adds the ‘ls’ listing and ‘na\n’. The
handoff between the two is still a bit murky to me. If the
uniplex-transport conversation is:
← /multistream/1.0.0
← /bird/3.2.1
← hey, how is it going?
Switching to a /bird/ protocol is a multistream-select situation. So
it should probalby be:
← /multistream-select/1.0.0
← /bird/3.2.1
← hey, how is it going?
And the listener handlers will be:
- Open with a multistream-select driver on the socket.
- Receive ‘/multistream-select/1.0.0’. Good, that's the language we
speak. - Receive ‘/bird/3.2.1’, that's a protocol we support. Pop ourselves
off the socket (and into a protocol stack?), and attach the
bird-3.2.1 driver. - Replay the ‘/bird/3.2.1’ message into the bird driver?
- Receive ‘hey, how is it going?’ and process with the current bird
driver.
The sticky bit is (4), which I don't think we actually need. But if
the ‘/bird/3.2.1’ is only sent on the multistream-select “stream”,
then the bird stream itself is just ‘hey, how is it going?’, which
isn't a multistream protocol. That's fine, it doesn't have to be a
multistream protocol, but if the only multistream protocol is
multistream-select, it seems a bit odd ;).
I'm also not sure how, in the uniplexed transport, a subprotocol is
supposed to open a new child protocol. For example, if the logicial
stream spawning structure was like:
← /multistream-select/1.0.0
← /bird/3.2.1
← /parrot/1.0.0
If you have a parallel stream that's still in multistream-select, you
can just ask for a new stream with the child protocol there:
← /multistream-select/1.0.0
← /bird/3.2.1
… bird stream …
← /parrot/1.0.0
… new parrot stream …
That means the agent requesting the opening needs to reach back up to
the multistream-select stream, but that's ok. Over uniplexed
transport, it seems like you'd either have to build subprotocol
spawning into your spawning protocol:
← /multistream-select/1.0.0
← /bird/3.2.1
… bird stream …
← /parrot/1.0.0 # handled by bird driver
… new parrot stream …
Or close the parent:
← /multistream-select/1.0.0
← /bird/3.2.1
… bird stream …
← exit # or whatever the bird-driver needs to shut down
← /parrot/1.0.0 # handled by multistream-select driver
… new parrot stream …
And I'm a bit nervous about both of those choices. Maybe that's just
stretching multistream-select over unplexed transport too far?
|
@diasdavid i'm going to merge this |
Add pdd protocol and multistream example as .md files for easier review and upgrade collaboratively
|
👍 :) |
Continuation of the work started here:
#11 (comment)